{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine Learning using Tensorflow\n",
    "\n",
    "In this notebook we will solve two simple exercises using low-level\n",
    "Tensorflow.\n",
    "\n",
    "Table of Contents:\n",
    "\n",
    "- [ 1 Iterative division](#1-Iterative-division)\n",
    "- [ 2 Neural function approximator](#2-Neural-function-approximator)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'1.4.1'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "from matplotlib import pyplot\n",
    "# Always make sure you are using running the expected version.\n",
    "# There are considerable differences between versions...\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i _derived/2_visualize_graph.py\n",
    "# (Load code helper code cell -- make sure to have executed notebook \"2_tf_basics\" first.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 Iterative division\n",
    "\n",
    "Using [stochastic gradient descent](https://en.wikipedia.org/wiki/Stochastic_gradient_descent)\n",
    "to iteratively solve a division:\n",
    "We first define our graph `A = X * B` and then we provide both\n",
    "`A` and `B`, and set an initial guess of `X` to zero. We then\n",
    "iteratively improve our guess for `X` by minimizing the square of the\n",
    "difference of `A` (=target) and `X * B`.\n",
    "\n",
    "Note that we do not need to specify any gradients -- Tensorflow's\n",
    "`GradientDescentOptimizer` can simply look at our \"forward graph\" and\n",
    "then generate a \"backward graph\" starting from our loss that consists\n",
    "of the gradients!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use new graph for this section to keep things tidy.\n",
    "graph = tf.Graph()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe seamless style=\"width:1200px;height:620px;border:0\" srcdoc=\"\n",
       "        <script>\n",
       "          function load() {\n",
       "            document.getElementById(&quot;graph0.920687133624&quot;).pbtxt = 'node {\\n  name: &quot;B&quot;\\n  op: &quot;Placeholder&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;X/initial_value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;X&quot;\\n  op: &quot;VariableV2&quot;\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;X/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;X&quot;\\n  input: &quot;X/initial_value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@X&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;X/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;X&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@X&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;A&quot;\\n  op: &quot;Placeholder&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;X/read&quot;\\n  input: &quot;B&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;A&quot;\\n  input: &quot;mul&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;pow/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sub&quot;\\n  input: &quot;pow/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;loss&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\n';\n",
       "          }\n",
       "        </script>\n",
       "        <link rel=&quot;import&quot; href=&quot;https://tensorboard.appspot.com/tf-graph-basic.build.html&quot; onload=load()>\n",
       "        <div style=&quot;height:600px&quot;>\n",
       "          <tf-graph-basic id=&quot;graph0.920687133624&quot;></tf-graph-basic>\n",
       "        </div>\n",
       "    \"></iframe>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Define our calculation + loss.\n",
    "\n",
    "with graph.as_default():\n",
    "    B = tf.placeholder(shape=(), dtype=tf.float32, name='B')\n",
    "    X = tf.Variable(0.0, name='X')\n",
    "    A = tf.placeholder(shape=(), dtype=tf.float32, name='A')\n",
    "    # Use overloaded Python operators for convenience.\n",
    "    # (this will translate to tf.mul(), tf.sub(), and tf.pow())\n",
    "    loss = (A - X * B) ** 2\n",
    "    # Let's use the identity Op to add a name to the \"loss\" tensor.\n",
    "    loss = tf.identity(loss, name='loss')\n",
    "\n",
    "    show_graph(tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe seamless style=\"width:1200px;height:620px;border:0\" srcdoc=\"\n",
       "        <script>\n",
       "          function load() {\n",
       "            document.getElementById(&quot;graph0.53663085152&quot;).pbtxt = 'node {\\n  name: &quot;B&quot;\\n  op: &quot;Placeholder&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;X/initial_value&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;X&quot;\\n  op: &quot;VariableV2&quot;\\n  attr {\\n    key: &quot;container&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;shared_name&quot;\\n    value {\\n      s: &quot;&quot;\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;X/Assign&quot;\\n  op: &quot;Assign&quot;\\n  input: &quot;X&quot;\\n  input: &quot;X/initial_value&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@X&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n  attr {\\n    key: &quot;validate_shape&quot;\\n    value {\\n      b: true\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;X/read&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;X&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@X&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;A&quot;\\n  op: &quot;Placeholder&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;shape&quot;\\n    value {\\n      shape {\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;X/read&quot;\\n  input: &quot;B&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;A&quot;\\n  input: &quot;mul&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;pow/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 2.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sub&quot;\\n  input: &quot;pow/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;loss&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Shape&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n          }\\n        }\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Const&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 1.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/Fill&quot;\\n  op: &quot;Fill&quot;\\n  input: &quot;gradients/Shape&quot;\\n  input: &quot;gradients/Const&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Shape&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n          }\\n        }\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Shape_1&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n          }\\n        }\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/pow_grad/Shape&quot;\\n  input: &quot;gradients/pow_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/Fill&quot;\\n  input: &quot;pow/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/sub/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 1.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/sub&quot;\\n  op: &quot;Sub&quot;\\n  input: &quot;pow/y&quot;\\n  input: &quot;gradients/pow_grad/sub/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Pow&quot;\\n  op: &quot;Pow&quot;\\n  input: &quot;sub&quot;\\n  input: &quot;gradients/pow_grad/sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/pow_grad/mul&quot;\\n  input: &quot;gradients/pow_grad/Pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/pow_grad/mul_1&quot;\\n  input: &quot;gradients/pow_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tidx&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/pow_grad/Sum&quot;\\n  input: &quot;gradients/pow_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tshape&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Greater/y&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Greater&quot;\\n  op: &quot;Greater&quot;\\n  input: &quot;sub&quot;\\n  input: &quot;gradients/pow_grad/Greater/y&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Log&quot;\\n  op: &quot;Log&quot;\\n  input: &quot;sub&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/zeros_like&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.0\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Select&quot;\\n  op: &quot;Select&quot;\\n  input: &quot;gradients/pow_grad/Greater&quot;\\n  input: &quot;gradients/pow_grad/Log&quot;\\n  input: &quot;gradients/pow_grad/zeros_like&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/mul_2&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/Fill&quot;\\n  input: &quot;pow&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/mul_3&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/pow_grad/mul_2&quot;\\n  input: &quot;gradients/pow_grad/Select&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/pow_grad/mul_3&quot;\\n  input: &quot;gradients/pow_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tidx&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/pow_grad/Sum_1&quot;\\n  input: &quot;gradients/pow_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tshape&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/tuple/group_deps&quot;\\n  op: &quot;NoOp&quot;\\n  input: &quot;^gradients/pow_grad/Reshape&quot;\\n  input: &quot;^gradients/pow_grad/Reshape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/tuple/control_dependency&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;gradients/pow_grad/Reshape&quot;\\n  input: &quot;^gradients/pow_grad/tuple/group_deps&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@gradients/pow_grad/Reshape&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/pow_grad/tuple/control_dependency_1&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;gradients/pow_grad/Reshape_1&quot;\\n  input: &quot;^gradients/pow_grad/tuple/group_deps&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@gradients/pow_grad/Reshape_1&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/Shape&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n          }\\n        }\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/Shape_1&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n          }\\n        }\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/sub_grad/Shape&quot;\\n  input: &quot;gradients/sub_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/pow_grad/tuple/control_dependency&quot;\\n  input: &quot;gradients/sub_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tidx&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/sub_grad/Sum&quot;\\n  input: &quot;gradients/sub_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tshape&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/pow_grad/tuple/control_dependency&quot;\\n  input: &quot;gradients/sub_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tidx&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/Neg&quot;\\n  op: &quot;Neg&quot;\\n  input: &quot;gradients/sub_grad/Sum_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/sub_grad/Neg&quot;\\n  input: &quot;gradients/sub_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tshape&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/tuple/group_deps&quot;\\n  op: &quot;NoOp&quot;\\n  input: &quot;^gradients/sub_grad/Reshape&quot;\\n  input: &quot;^gradients/sub_grad/Reshape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/tuple/control_dependency&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;gradients/sub_grad/Reshape&quot;\\n  input: &quot;^gradients/sub_grad/tuple/group_deps&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@gradients/sub_grad/Reshape&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/sub_grad/tuple/control_dependency_1&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;gradients/sub_grad/Reshape_1&quot;\\n  input: &quot;^gradients/sub_grad/tuple/group_deps&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@gradients/sub_grad/Reshape_1&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/Shape&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n          }\\n        }\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/Shape_1&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_INT32\\n        tensor_shape {\\n          dim {\\n          }\\n        }\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/BroadcastGradientArgs&quot;\\n  op: &quot;BroadcastGradientArgs&quot;\\n  input: &quot;gradients/mul_grad/Shape&quot;\\n  input: &quot;gradients/mul_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/mul&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;gradients/sub_grad/tuple/control_dependency_1&quot;\\n  input: &quot;B&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/Sum&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/mul_grad/mul&quot;\\n  input: &quot;gradients/mul_grad/BroadcastGradientArgs&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tidx&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/Reshape&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/mul_grad/Sum&quot;\\n  input: &quot;gradients/mul_grad/Shape&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tshape&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/mul_1&quot;\\n  op: &quot;Mul&quot;\\n  input: &quot;X/read&quot;\\n  input: &quot;gradients/sub_grad/tuple/control_dependency_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/Sum_1&quot;\\n  op: &quot;Sum&quot;\\n  input: &quot;gradients/mul_grad/mul_1&quot;\\n  input: &quot;gradients/mul_grad/BroadcastGradientArgs:1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tidx&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n  attr {\\n    key: &quot;keep_dims&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/Reshape_1&quot;\\n  op: &quot;Reshape&quot;\\n  input: &quot;gradients/mul_grad/Sum_1&quot;\\n  input: &quot;gradients/mul_grad/Shape_1&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;Tshape&quot;\\n    value {\\n      type: DT_INT32\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/tuple/group_deps&quot;\\n  op: &quot;NoOp&quot;\\n  input: &quot;^gradients/mul_grad/Reshape&quot;\\n  input: &quot;^gradients/mul_grad/Reshape_1&quot;\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/tuple/control_dependency&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;gradients/mul_grad/Reshape&quot;\\n  input: &quot;^gradients/mul_grad/tuple/group_deps&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@gradients/mul_grad/Reshape&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;gradients/mul_grad/tuple/control_dependency_1&quot;\\n  op: &quot;Identity&quot;\\n  input: &quot;gradients/mul_grad/Reshape_1&quot;\\n  input: &quot;^gradients/mul_grad/tuple/group_deps&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@gradients/mul_grad/Reshape_1&quot;\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;GradientDescent/learning_rate&quot;\\n  op: &quot;Const&quot;\\n  attr {\\n    key: &quot;dtype&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;value&quot;\\n    value {\\n      tensor {\\n        dtype: DT_FLOAT\\n        tensor_shape {\\n        }\\n        float_val: 0.10000000149\\n      }\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;GradientDescent/update_X/ApplyGradientDescent&quot;\\n  op: &quot;ApplyGradientDescent&quot;\\n  input: &quot;X&quot;\\n  input: &quot;GradientDescent/learning_rate&quot;\\n  input: &quot;gradients/mul_grad/tuple/control_dependency&quot;\\n  attr {\\n    key: &quot;T&quot;\\n    value {\\n      type: DT_FLOAT\\n    }\\n  }\\n  attr {\\n    key: &quot;_class&quot;\\n    value {\\n      list {\\n        s: &quot;loc:@X&quot;\\n      }\\n    }\\n  }\\n  attr {\\n    key: &quot;use_locking&quot;\\n    value {\\n      b: false\\n    }\\n  }\\n}\\nnode {\\n  name: &quot;GradientDescent&quot;\\n  op: &quot;NoOp&quot;\\n  input: &quot;^GradientDescent/update_X/ApplyGradientDescent&quot;\\n}\\n';\n",
       "          }\n",
       "        </script>\n",
       "        <link rel=&quot;import&quot; href=&quot;https://tensorboard.appspot.com/tf-graph-basic.build.html&quot; onload=load()>\n",
       "        <div style=&quot;height:600px&quot;>\n",
       "          <tf-graph-basic id=&quot;graph0.53663085152&quot;></tf-graph-basic>\n",
       "        </div>\n",
       "    \"></iframe>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Magic happens : add optimizer.\n",
    "\n",
    "with graph.as_default():\n",
    "    optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.1)\n",
    "    # We can now minimize the loss by calling this train_op (many times),\n",
    "    # which will change the value of \"X\" upon every invocation a bit in such\n",
    "    # a way that the loss becomes smaller.\n",
    "    train_op = optimizer.minimize(loss)\n",
    "\n",
    "    show_graph(tf.get_default_graph())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X= [0.0]\n",
      "X= [16.800001]\n",
      "X= [20.16]\n",
      "X= [20.832001]\n",
      "X= [20.9664]\n",
      "X= [20.99328]\n",
      "X= [20.998655]\n",
      "X= [20.999731]\n",
      "X= [20.999947]\n",
      "X= [20.999989]\n",
      "X= [20.999998]\n"
     ]
    }
   ],
   "source": [
    "# Do the computation.\n",
    "\n",
    "with tf.Session(graph=graph) as sess:\n",
    "    # Variables must be initialized before first use.\n",
    "    tf.global_variables_initializer().run()\n",
    "    # Feed the same result at every step.\n",
    "    feed_dict = {B: 2, A: 42}\n",
    "\n",
    "    print 'X=', sess.run([X])\n",
    "    for i in range(10):\n",
    "        # Update b by calling train_op.\n",
    "        sess.run([train_op], feed_dict=feed_dict)\n",
    "        # Print our updated guess for X.\n",
    "        print 'X=', sess.run([X])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 Neural function approximator"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Neural networks can be seen as generic function approximators\n",
    "`y = f(x)` -- in this section we will use a network to approximate\n",
    "a simple mathematical function with a scalar input.\n",
    "\n",
    "The same approach can be used to train arbitrary function (for example\n",
    "a function where `x` is a dense vector of a megapixel image, and `y` is\n",
    "a probability distribution over thousands of different object classes).\n",
    "\n",
    "A crucial part when designing such a network is a good choice of\n",
    "\"hyperparameters\" that define *how* the network should be created and\n",
    "trained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper class for function approximation. Defining all the code\n",
    "# in a single class makes it easier to instantiate different\n",
    "# \"versions\" of the approximator (to approximate different functions\n",
    "# with different configurations).\n",
    "\n",
    "class FunctionApproximator(object):\n",
    "    \"\"\"A class for function approximation using Tensorflow.\n",
    "\n",
    "    This class implements both building the graph and updating the\n",
    "    variables to better approximate the target function, so that it\n",
    "    can be instantiated with different functions/configurations.\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, f, hidden_layers):\n",
    "        \"\"\"Initializes a new function approximator.\n",
    "        \n",
    "        Args:\n",
    "          f: Scalar Python functional that should be approximated.\n",
    "            Signature: f(x:Number) -> Number\n",
    "          hidden_layers: Iterable specifying number of units per layer.\n",
    "        \"\"\"\n",
    "        self.f = f\n",
    "        self.hidden_layers = hidden_layers\n",
    "        # Initialize new graph that only contains Ops needed for\n",
    "        # function approximation.\n",
    "        self.graph = tf.Graph()\n",
    "        with self.graph.as_default():\n",
    "            self._build()\n",
    "        self.init()\n",
    "\n",
    "    def init(self):\n",
    "        \"\"\"Initializes weights.\"\"\"\n",
    "        self.sess = tf.Session(graph=self.graph)\n",
    "        self.init_op.run(session=self.sess)\n",
    "\n",
    "    def _build(self):\n",
    "        # The values for \"x\" and \"y\" are provided during training.\n",
    "        # Note the dynamic first dimension (=number of samples in batch).\n",
    "        self.x = tf.placeholder(shape=(None,), dtype=tf.float32)\n",
    "        self.y = tf.placeholder(shape=(None,), dtype=tf.float32)\n",
    "        # Reshape e.g. [1, 2, 3] to [[1], [2], [3]] because this latter\n",
    "        # shape is required by tf.layers() below.\n",
    "        x = tf.reshape(self.x, (-1, 1))\n",
    "        for units in self.hidden_layers:\n",
    "            # We use tf.layers() helper function to create \"fully connected\n",
    "            # layers\" where every neuron is connected to every neuron in the\n",
    "            # previous layer. This helper function takes care of defining\n",
    "            # variables and initializing them.\n",
    "            x = tf.layers.dense(inputs=x, units=units, activation=tf.nn.relu)\n",
    "        # Our predicted \"y\".\n",
    "        self.y_ = tf.layers.dense(inputs=x, units=1, activation=None)\n",
    "        # Reshape e.g. [[1], [2], [3]] (by tf.layers()) to [1, 2, 3].\n",
    "        self.y_ = tf.reshape(self.y_, (-1,))\n",
    "\n",
    "        # Compute loss.\n",
    "        self.loss = tf.reduce_mean((self.y - self.y_)**2)\n",
    "        # Add Ops for minimizing loss.\n",
    "        optimizer = tf.train.GradientDescentOptimizer(learning_rate=0.01)\n",
    "        self.train_op = optimizer.minimize(self.loss)\n",
    "        self.init_op = tf.global_variables_initializer()\n",
    "\n",
    "    def fit(self, x_min, x_max, steps, batch_size, show_progress=False):\n",
    "        \"\"\"Fits the weights to better approximate the function.\n",
    "\n",
    "        This function will generate random samples \"x\" within [x_min, x_max]\n",
    "        and update the network's weight to better approximate y=f(x).\n",
    "\n",
    "        Args:\n",
    "          x_min: Lower bound for generation of \"x\".\n",
    "          x_max: Upper bound for generation of \"x\".\n",
    "          steps: Number of training steps.\n",
    "          batch_size: Number of samples to train network with in every step.\n",
    "        \"\"\"\n",
    "        t0 = time.time()\n",
    "\n",
    "        losses = []\n",
    "        for step in range(steps):\n",
    "            # Generate samples.\n",
    "            x_data = np.random.uniform(low=x_min, high=x_max, size=batch_size)\n",
    "            y_data = self.f(x_data)\n",
    "            feed_dict = {\n",
    "                self.x: x_data,\n",
    "                self.y: y_data\n",
    "            }\n",
    "            # Update the weights and get the loss.\n",
    "            loss_data, _ = self.sess.run([self.loss, self.train_op], feed_dict=feed_dict)\n",
    "            losses.append(loss_data)\n",
    "            if show_progress and step % (steps // 10) == 0:\n",
    "                print 'step=%6d loss=%f' % (step, loss_data)\n",
    "\n",
    "        dt = time.time() - t0\n",
    "        return losses, dt\n",
    "\n",
    "    def predict(self, x_data):\n",
    "        \"\"\"Computes approximated function values.\"\"\"\n",
    "        return self.sess.run([self.y_], {self.x: x_data})[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step=     0 loss=0.625204\n",
      "step=  1000 loss=0.023981\n",
      "step=  2000 loss=0.006048\n",
      "step=  3000 loss=0.002083\n",
      "step=  4000 loss=0.001016\n",
      "step=  5000 loss=0.000465\n",
      "step=  6000 loss=0.000424\n",
      "step=  7000 loss=0.000253\n",
      "step=  8000 loss=0.000179\n",
      "step=  9000 loss=0.000124\n",
      "28.31 seconds\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f2f6c773a50>]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmcjXX7wPHPZcwwtsYytpEl+07PZElUdgmTFkSpX6WSJDWWUuQho1FI6klaFFkqTYoaCmWNYTCWxr6dEVMMYYxZvr8/zuKM7OfMnGWu9+t1Xs79ve/7nOtE5zrfXYwxKKWUUnb5PB2AUkop76KJQSmlVDaaGJRSSmWjiUEppVQ2mhiUUkplo4lBKaVUNpoYlFJKZaOJQSmlVDaaGJRSSmWT39MB3IhSpUqZypUrezoMpZTyKRs2bPjLGBN6tet8MjFUrlyZuLg4T4ehlFI+RUQOXMt12pSklFIqG00MSimlstHEoJRSKhtNDEoppbLRxKCUUiobtyQGEflERI6JyNbLnBcReVdEdovIFhG51elcXxHZZXv0dUc8Simlbpy7agyfAR2vcL4TUN326Ad8ACAiJYCRQFOgCTBSRIq7KSallFI3wC3zGIwxv4lI5Stc0g343Fj3EV0rIiEiUg64C1hijDkOICJLsCaY2e6ISynlXWLiLUTHJpKUkspNwYGIQMrZdMqHBBPZoSYRjcOu6zWu5z517XJrglsYcMjp+LCt7HLlSik/ExNvYfj8BFLTMwFISU13nLOkpDJ8fgLAFb/kL36Na71PXR+fmfksIv2wNkNRsWJFD0ejlLpW9l/4lpRUAIqfPUmrfRu5Y/9mmh1KoND5VNID8pMeEEjWh0FQNgSCgqyPAgUuPA8KotCeFP6bKZwPyE9C2ep8U68NqUB0bKImBjfKrcRgAW52Oq5gK7NgbU5yLl9+qRcwxkwDpgGEh4ebnAhSKeVe9l/4RU4k02fnGjrtXEXTg1vJb7I4UbAoqys14HihmwjMzCAwM52gzAwqVS0J589feJw86XhePekEgZnpBKen8fDmWJ5bM5cpt/dkfr02nv6ofiW3EsMCYICIzMHa0XzSGHNERGKBN506nNsDw3MpJqWUm13c/l/g+F8MWTqT3vE/EpSVwe4SFfig2YMsqd6UhLLVMJJ9/EtYSDBdhrW+7Ov3iVpqrXkYQ6t9Gxm8cibjf5rC8+u+gfonoXdvCAjI6Y/p99ySGERkNtZf/qVE5DDWkUaBAMaY/wGLgHuA3cBZ4HHbueMi8l9gve2lRts7opVSviMm3sKoBdsc/QaFzqdy/8LZ9Fs3n4Lpacxt2J5P/9OV3aUu3wwcHBhAZIeaV3yfyA41HX0Mv93yH36rciud9scxbvPX0LcvvPkmjBoFDz1EzOYj2kl9g8Q6UMi3hIeHG11dVSnv0PujNazaY/09JyaLXptjeXHlLELPpPBjjduJbvUoe0tW+Nd9Ie4cldSwHMTEwMiRsHUrJ6vV5LVGD7LglqYgAlgTz7ju9fN0chCRDcaY8Ktep4lBKXUjrP0HW0hNzwIg7OQxJiyaSPODCayrUIeoux5nY1jtf92Xo1/QWVkwbx4HBg6hUvIhtpapyrCOz7O1bDXHJWF5uPZwrYnBZ0YlKaW8x4iYBGatPYj9Z2XX7csZE/s+giGy0wt8Vb+t45d6SHAghQvkz50mnXz5oGdP2mwsRNfty4n89XPmf/EyE1o9wkdN7sNIPh3ieg00MSilrktMvIWZaw8CUCTtLKOXfED3bcuIC6vNi/e+xKGQso5rgwMDGNW1bq5/AZcpUYT59drwS9UmRP00hVeWf2rtrO48mGNFS5KanqlDXK9AF9FTSl2zETEJDJq7CYCGSYks+vR5um7/lXfu6E2Ph6OyJYXihQI91qYf2aEmwYEBnAwuyrMRwxna8XluTfqDHz99ntv3W+NPss2rUP+mfQxKqWsyIibBUVN4IOFn3vzpPY4VKcHALpFsrJC9L6FF1RLMeqq5J8J0uHhiXdW/D/HBt+Ooevwwb935KNObPkAW5KkRS9r5rJRym5h4i7WmYAwvrvySF1bPZmWlhjzXbRgng4s6rssn8HDTioyJqO/BaLNzXkaj0PlU3lo0mXsTV/JjjduJvGcQpwsUyjMjljQxKKXcwv7Feu58Oq//8hGPb/ieefXb8kqHAWQEXOimDA7Mx47/dvJgpJfnPMQ1H/D4uvkMW/4Z+0qE8fgDIzkcUpawkGBWXWFynT/QxKCUcosWUUtJOnGGqB+n0CNhCR/dFsHYu59wjDoCCMwnRD/Y0Cd+cVcZthADND+whQ9i3iQjXwBPdX+N+LBaCP7dtHStiUE7n5VSV3TkxBnG/fQePRKW8G7zHv9KCoWDAnwmKYD1ix9gTaUGdO8zgTNBwcye8wqd/liJwbpi64tzNzEiJsGzgXqQJgal1L/ExFtoEbWUW4Z+z5uxU+m5ZTHvNu/BOy37ZEsKfZpVZNvojj6TFODCiCWAvSUrcN8jb7O1TFU++C6Kvhu+B8AAs9YeJCbe4sFIPUcTg1IqG3ufguXEWUYv/oCem2N5r/lD2ZJCcGAAk3o08qpO5msV0TiMcd3rExYSjADHC91E755j+alGc974+UPuT/gFsCaHUQu2eTRWT9HEoJTKJjo2kdT0TIYt/5Q+m37k/WYPMKHlIwTky4dgXVLC10fwRDQOY9Ww1uyL6kxYSDBp+YMY2GUIKyo1YvyPk7lz7wbAuplQXqw1aGJQSmWTlJJKn40LeWbdfGbc2pm3WvUFEbKMYV9UZ1YNa+3TSeFikR1qIsD5/IE8e98rJIZW5r3voqh83JoQomMTPRugB2hiUEpl8+CReN74+UOWVGvCG236OZqP7J22/iaicRi9m1mXAz9doBBP3v8aGfny896CtyiQcR5LSiotopbmqZqDJgal1AVxcbw57022l63KwC5DyMpn7aS9lr0SfNmYiPoULxQIwJFiobzU+UXqHd3DK8s+Bi7sLZ1XkoMmBqWU1f79cO+95C9TmsMz5lKidHG/6VO4FiO71HWMVlparQnTbruPvhsX0umPlQCkpmfy0rzNeSI5uGsHt47AZCAAmG6Mibro/ETgbtthIaC0MSbEdi4TsA8YPmiM6eqOmJRS1+HECbjnHkhLg2XL6FS7Np3aeTqo3GVPfPb1ld66sy+3Hd5O1E9T2Fy+BknFSpNpTJ5YstvlGoOIBABTgU5AHaCXiNRxvsYY86IxppExphEwBZjvdDrVfk6TglK5Kybewl1jfmJNoztJ37mLFdEfQe1/b66TV9hHK4WFBJMRkJ8XurxMgMli4g/vkC8rE8CxZLc/c0dTUhNgtzFmrzHmPDAH6HaF63sBs93wvkopF8TEWxj+zRYGznmL5gcTeLnTC/Q7UCRPNJVcjX0S3MHi5Xi93TM0PbSVp9dd+D3r70t2uyMxhAGHnI4P28r+RUQqAVWApU7FBUUkTkTWikjE5d5ERPrZrotLTk52Q9hK5W3RsYn0WBtD923LiG75CN/VvTtP/Bq+FvZJcAEizK/bmh9q3sGglbOokbwf8N8RWna53fncE/jaGJPpVFbJtqjTw8AkEal6qRuNMdOMMeHGmPDQ0NDciFUpv3ZT4jaGL/+EJdWaMLX5Q45yf/81fK0iGofx9kMNCQ7Kz+vtn+WfAoWZsGgS+TMzsKSkUnnYQhqPXuyXNSx3JAYLcLPTcQVb2aX05KJmJGOMxfbnXmA50NgNMSmlruTMGd5fOIETwcUY0umFbOsf+fuv4ethrzkEly/LiPb9afDnbp75/WvH+RNn0xk8b5PfJQd3JIb1QHURqSIiQVi//BdcfJGI1AKKA2ucyoqLSAHb81JAC2C7G2JSSl3JoEFU+usQQyOGcKLQTY5if5+vcCPsHdIJTduyoHYrBq6aQ+1jex3ns4z/rankcmIwxmQAA4BYYAcwzxizTURGi4jzKKOewByTfQOI2kCciGwGlgFRxhhNDErlpHnzYPp0ZPhw7hvcx7GYXF6Zr3CjklJSeb3dM5wMLsLbCycSmJnuOJeSmn6FO32PbtSjVF6yfz80agS1asGKFRAY6OmIfEaLqKVYUlJpt2stH80fw6QWvZh0R2/H+Uk9Gnl9UtWNepRS2WVkQO/eYAzMnq1J4TrZm9iWVG/Gt3Xu4rk186h7dI/jvD+N5tLEoJSfs2+68+6dj8Dq1awf9iZUqeLpsHxOROMw+tgW2xvV9mlOBBdjglOTkj8ttqeJQSk/Zt905+YtvzNg9Vy+qteWR89W9YsvL0+wL7Z3MrgowzsOoHbyfgasnuc47y+L7WliUMqPRccmEvjPSSZ+/zb7i5djZLundRKbi+yL7f1SrSnf1L2b59bMpe6fux3n/eG/ryYGpfxYUkoqw5d9QukzJxjU5WXOBgU7ytWNcd4a9I22T/N34RAmLJqUbZSSJSXVp2sNmhiU8mNd/tpBry2L+ajJfWwpV8NRrpPYXGOf21C0bCjDOj5P7eT9PL9qTrZrfLlJSRODUv7qzBnGxb7HgeLlmdjiYUexTmJzn8gONVlbqxlf12tD/7Vf/atJyVcnvmliUMpfvfYahQ8f4OBb71IqNEQnseUAe7PS6DZP8dclmpRSUtN9stagE9yU8kdr18Ltt8Mzz8D773s6Gr/XImopNTb8xqdfv8G7zXvwTqtHHOfCQoJZNay1B6O7QCe4KZVXpaXBE09AWBhERV39euWyyA41WVb1Nr6p1/pfTUq+2NGviUEpP2GfyDa59eOwfTtrIsdCsWKeDitPiGgcRvFCgbzRpt+/Rin5Yke/Jgal/IB9IlvRndvpv3Ye39a5i/9LLu2T7du+amSXuqQXvYlXOjznmPjmqx39mhiU8gPRsYmcTztP1E/vcrJgEUa3ecovJlr5EntH9B/hd/Ft3bt5bu08BhQ/RXRsIlWGLfSp5TI0MSjlB5JSUnlk40IaHdnFG236OfZY8MX2bV9mn99w329fk1GiJG3GD+PY36cw+NZyGZoYlPIDDfKd4aUVX/BrlVv5vnYrR7kvtm/7hRIlGNnpeWod28dzay6speQrtTi3JAYR6SgiiSKyW0SGXeL8YyKSLCKbbI8nnc71FZFdtkdfd8SjVF4zdeOXBGVm8Hq7ZxzbdPpq+7a/mFe+MfPr3s1za+ZR5+iFHd98oRbncmIQkQBgKtAJqAP0EpE6l7h0rjGmke0x3XZvCWAk0BRoAowUkeKuxqRUnvLzz1SI/Y69Tz5PRpWqOpHNS5QPCbY26wUXY8KiiT41SskdNYYmwG5jzF5jzHlgDtDtGu/tACwxxhw3xpwAlgAd3RCTUnlDWho89xxUq0btSWNZNaw1+6I6s2pYa00KHhbZoSbni4XwSocB1Dm2j/5rvkLwjX0b8rvhNcKAQ07Hh7HWAC52v4i0AnYCLxpjDl3mXv3XrNS1io6GnTshNhYKFvR0NMqJPTFHFw7i2z9WMGDNXBbXaMaO0rc4OqKdr/MmudX5/D1Q2RjTAGutYMb1voCI9BOROBGJS05OdnuASvmcvXth7Fh48EFo397T0ahLsI9SmtZ9ICnBRZmwcBL5MzMA7+6IdkdisAA3Ox1XsJU5GGP+Nsak2Q6nA/+51nudXmOaMSbcGBMeGhrqhrCV8mHGwPPPQ/78MHGip6NRV/FHehCvtn+Ousf20n/tV45yb+2IdkdiWA9UF5EqIhIE9AQWOF8gIuWcDrsCO2zPY4H2IlLc1unc3lamlLqSmBhYtAhGj7auiaS8WvmQYBbXaM53te/k+dVzqH3MOkrJgFf2N7icGIwxGcAArF/oO4B5xphtIjJaRLraLhsoIttEZDMwEHjMdu9x4L9Yk8t6YLStTCl1OadPwwsvQIMG1lqD8nqRHWoSHBjAqLb9SClYlOhFkx1NSt448U2X3VbK1wwZYu10XrkSWrTwdDTqGsXEW4iOTaTeul/48Ns3efuO3kxp0ctxPjeW59Zlt5XyR1u3WvsUnnhCk4KPsXdEL65xOwtqt+L51XOpdWyf47w39TdoYlDKVxgD/ftbl9LWfRZ8VvmQYEa2fZqTBYswYdGFUUreNPFNE4NSvuLzz2HFChg/HkqV8nQ06gZFdqjJuZtKMKJ9f+od3cMzv3/tdcuXuGOCm1Iqpx0/DpGR0Lw5/N//eToa5QLniW/f/7GCF1bN4ff6d/DiXOu8hsgONT0+6U1rDEr5gldftSaHDz6AfPq/ra+z9zcETJ3KqYKFef2baAIyM7xmhJL+C1PK261bBx9+CAMHQsOGno5GudHYdcm82r4/9W1NSmCdET1qwTaPxqWJQSkvFRNvoeWbS9ja9WGSi5bgh/v6eTok5WZJKan8VLMFP9RqycBVc6iZvB+AlNR0j9YaNDEo5YVGxCTw4txN3L3sG+od3cOou54kcvF+jzcxKPeyj0R6vd0znCpYmAkLJzpGKXlyHSVNDEp5mZh4C7PWHqRY6j8MXjmLFZUasbDWHV696Jq6MfaRSMcL3cRr7Z6l/tE99Fs3H/DsvAZNDEp5mejYRAwwYM1ciqadZUybJx27snnTJCjluojGYRQvFAjAj7Xu4IdaLRm08ktqJO/36LwGTQxKeRlLSirlTx3j0Y0/8E291iSGVnac86ZJUMo9RnapS3BgAGBtUvqnQCHe/nEyQ9pU9VhMmhiU8iIjYqybt7y44ktAmHhHb8c5Aa+aBKXcI6JxGOO61ycsJJgThW5i4n2DqH9kF91+/tJjMekEN6W8hL1voXryAbpvW8on4V05UuzC3iO9m1X0+MQnlTMiGoc5/d12hvQdMGoUdO0KdevmejxaY1DKS9j7FiJXfMGZwIJMbf5QtvNjIup7JjCV+957z7om1uOPQ0ZGrr+9JgalvIQlJZVbLTtov2st05p2JyW4mONcmPYt5C2lS8P778P69fD227n+9poYlPICMfEWxBiG/jqD5MIhfBwe4TinfQt51IMPwgMPwOuvw/btufrWbkkMItJRRBJFZLeIDLvE+cEisl1EtojILyJSyelcpohssj0WXHyvUnlBdGwirfZtpOmhrbx7e09Sgwo6zmnfQh42daq1Semxx3K1ScnlxCAiAcBUoBNQB+glInUuuiweCDfGNAC+Bt5yOpdqjGlke3RFqTzoyIkzDP31Mw6ElGVOww7ZzmnfQt4VY0nntXbPwvr1TO3SP9dmvrujxtAE2G2M2WuMOQ/MAbo5X2CMWWaMOWs7XAtUcMP7KuU3HjnwO3WO7ePtln1IDwh0lGvfQt4VE29h+PwEvri5CYtq3M6TSz7jo2kLcyU5uCMxhAGHnI4P28ou5wngR6fjgiISJyJrRSTicjeJSD/bdXHJycmuRayUNzl/niGrZvJH6Sp8X7uVo9jbNm9RuSs6NpHU9EwQ4fX2z3ImKJjXFk7hpXmbczw55Oo8BhHpA4QDdzoVVzLGWETkFmCpiCQYY/ZcfK8xZhowDSA8PNzkSsBK5YaPP6bw4QOceHcG5c8UJikllfIhwV6xYYvyHOflT/4qXJyBXSJJKhZKpjEMn2+dCJlT/z7ckRgswM1OxxVsZdmISFvgVeBOY0yavdwYY7H9uVdElgONgX8lBqX80pkzMHo0tGxJ8wGPsMq2JpJS5UOCsTglh5VVGjue2xdUzKnE4I6mpPVAdRGpIiJBQE8g2+giEWkMfAh0NcYccyovLiIFbM9LAS2A3B2XpZQnvfsu/PknjBvnWChPKbAOUbavoXQpObmgoss1BmNMhogMAGKBAOATY8w2ERkNxBljFgDRQBHgK7H+4z9oG4FUG/hQRLKwJqkoY4wmBpU3HD8O48dDly7QooWno1Fexl4beGneZjLNv1vPc3JBRbf0MRhjFgGLLip73el528vctxrQsXgqbxo/Hk6dgrFjPR2J8lL25DB8foK1I9ompwcm6CJ6SnmCxWJtRurTB+rrbyN1efbkEB2bmGsDEzQxKOUJo0dDZia88YanI1E+IPvqqzlPE4NSuSQm3kJ0bCIF9u5m8cfTOfBQX6pWqeLpsJT6F11ET6lcYJ/FaklJZfCKmaQFBNG3bLtcW+JAqeuhiUGpXGCfxVrvz93c+8cKpt8WweECxYiOTfR0aEr9iyYGpXKBfcx55G+fczy4GNOb3JetXClvoolBqVxQPiSY2w5t5c59G/mg6QP8U6Cwo1wpb6Odz0rlgsj2Najw/gscLVKCL269B9BF8pT30sSgVA6yj0SqtnElMw5tZdw9z5EWWJAwXSRPeTFNDErlEPtIpNTzGby/YiaHi5VmdsMOTHywkSYE5dW0j0GpHGIfidRh1xoa/rmLyS16cSorn45EUl5PE4NSOSQpJRUxWby4YhZ7SoQxv15rR7lS3kwTg1I5pHxIMJ3/WEmtvw4wqcXDZOYLcJQr5c20j0GpHBLZthr1JzxGYqmKLKx1B6AjkZRv0BqDUjkkInEFVf8+xOftH8PkCyAsJJhx3etrx7Pyem6pMYhIR2Ay1o16phtjoi46XwD4HPgP8DfQwxiz33ZuOPAEkAkMNMbEuiMmpTxlREwC89bs56ePhpAWWoV893dnX/eGng5LqWvmco1BRAKAqUAnoA7QS0TqXHTZE8AJY0w1YCIw3nZvHaxbgdYFOgLv215PKZ80IiaBmWsP0mXrMm45kcTEOx7mi3WHGRGT4OnQlLpm7mhKagLsNsbsNcacB+YA3S66phsww/b8a6CNWPf47AbMMcakGWP2Abttr6eUT5r9+yHyZ2YwcPVsEspUZXH1Zo5ypXyFOxJDGOD8r/6wreyS1xhjMoCTQMlrvFcpnxATbyHTGB5M+JlKKX8y8Y7eYN3j/JJ79irlrXym81lE+olInIjEJScnezocpbKxz3IukHGe51fPYWP5miytepvjfIAtQSjlC9yRGCzAzU7HFWxll7xGRPIDN2HthL6WewEwxkwzxoQbY8JDQ0PdELZS7mOf5dw7/kfK//MX0a0eddQWAHo1vfkKdyvlXdyRGNYD1UWkiogEYe1MXnDRNQuAvrbnDwBLjTHGVt5TRAqISBWgOrDODTEplauSUlIpnHaW59bMZVWlBqypdGEUUp9mFRkTUd+D0Sl1fVwermqMyRCRAUAs1uGqnxhjtonIaCDOGLMA+Bj4QkR2A8exJg9s180DtgMZwHPGmExXY1Iqt5UPCeb+hbMpmXqK8Xc+5igPCwnWpKB8jlvmMRhjFgGLLip73en5OeDBy9w7FhjrjjiU8pRXm5Si1cj5LKpxO1vK1QB0lrPyXbokhlJucM8Pn2Ey0pjZ+UkEaw1C91tQvkoTg1KuOngQpk5FHnuML9953NPRKOUynxmuqpTXGjXKOgJp5EhPR6KUW2hiUMoVO3bAjBnw3HNQsaKno1HKLTQxKOWKESOgcGEYPtzTkSjlNpoYlLpR69bB/PkQGQmlSnk6GqXcJk91PsfEW4iOTSQpJVVHjSjXGAPDhkFoKAwa5OlolHKrPJMY7GvZpKZb589ZUlIZPt+6FLImB3Xdfv4Zli2DyZOhaFFPR6OUW+WZpiT7WjbOUtMziY5N9FBEymcZY+1TqFQJnn7a09Eo5XZ5psaQlJJ6XeVKXdY338CGDdbRSAUKeDoapdwuz9QYyocEX1e5UpeUkQGvvgp160Lv3p6ORqkckWcSQ2SHmgQHZt81VNeyUdfts89g5054800I0F1olX/KM01JEY3DICuL9e9MZ3aFcMoVL6yjktT1SU21znJu3hy6dPF0NErlmDyTGAAiDm8k4svRjH3vPetMVaWux9SpYLHArFnZNuFRyt/kmaYkALp2hY4dYcgQ2L3b09EoX3LyJIwbZ/33c+edno5GqRyVtxKDCHz0EQQGwmOPQabuCaSuLCbeQouopUzp9DQcP86yR3Uym/J/LiUGESkhIktEZJftz+KXuKaRiKwRkW0iskVEejid+0xE9onIJtujkSvxXJMKFWDKFFi1CiZOzPG3U75rREwCL87dRJoliSfiYlhQuxX9txti4i+5LblSfsPVGsMw4BdjTHXgF9vxxc4Cjxpj6gIdgUkiEuJ0PtIY08j22ORiPNemTx/o1s26ANr27bnylsq3xMRbmLX2IAYYsHougZkZvN2yj06KVHmCq4mhGzDD9nwGEHHxBcaYncaYXbbnScAxINTF93WNCHz4IRQpAn378t36A7SIWkqVYQtpEbVUfxEqomMTMUClE0k8vOkn5jVox4Hi5QGdFKn8n6uJoYwx5ojt+Z9AmStdLCJNgCBgj1PxWFsT00QRuew0UhHpJyJxIhKXnJzsYthAmTLwwQcQF8f+yNexpKRisK6h9OLcTYyISXD9PZTPsn/5v7rsE87nD2RSiwuT2XRSpPJ3V00MIvKziGy9xKOb83XGGAOYK7xOOeAL4HFjTJateDhQC7gNKAEMvdz9xphpxphwY0x4aKibKhwPPsiSBnfTf8Us6hzde+G9gJlrD2rNIQ8rHxLM7fs30X7XWqY2f4jkItbuMwGdFKn83lUTgzGmrTGm3iUe3wFHbV/49i/+Y5d6DREpBiwEXjXGrHV67SPGKg34FGjijg91PYbc9RQngovx9sJ3CMpIz3Zu1IJtuR2O8hKRbasxctl0Dt1Uhk/Crb+BBOjdrKJOilR+z9WmpAVAX9vzvsB3F18gIkHAt8DnxpivLzpnTyqCtX9iq4vxXLdC5cowvOMAaifvZ+Dq2dnOpaSmX+Yu5a/sw1PXjYim5rH9vNvhKc7nDyIsJJiJPRoxJqK+p0NUKse5mhiigHYisgtoaztGRMJFZLrtmoeAVsBjlxiWOktEEoAEoBQwxsV4rltkh5r8Uq0pX9Vry7Nrv6ZRUvYRJ9qclHfY9+w4dfQvBq/4gt8r1OWH6rczsUcjVg1rrTUFlWeItWvAt4SHh5u4uDi3vV7d138i3z+n+OnjAZwLLMA9j00mLdDaDx6YT4h+sKF+KeQBLaKWWjdwWvYJT637lq59J7K1bDXCQoJZNay1p8NTymUissEYE3616/LWzOfLGHtffc4VKsKQe16g6vHDvLziC8e59CyjfQ15RFJKKlWOW3g8bgFf12/D1rLVHOVK5SV5ahG9y7HXBgbNNXzeuDNPrP+OxdWbsf7meoD2NeQV5UOCGfHqXRFkAAAWhUlEQVT1dNLyBxLdqm+2cqXyEq0x2NiTQ9Rdj3EopAwTFk2i0Hn9pZiXRBex0GbPet69vZdjeKru2aHyIk0MTooXCuRsUDAv3zOIm1OOMmz5Z45y5efOn+f2KWM4XbEKi9s8hABhIcGM615f+5dUnqNNSU5GdqlL5NebWX9zPT4J78qTcd8RW6M5qyo3okXUUt3Yxw/FxFuIjk2k0+IvGbFzJwmTZ/DrwA6eDkspj9Iag5OIxmFEP9CQsJBgols9yp4SFXhr0WSKpp2xjlaZn6DDV/2IfXhqmiWJgatms+yW//B/yaX171jleZoYLhLROIxVw1pTKjSElzq/SNnTf/PaLx8B6MqafiY6NpHU9Ewif/2c4Iw0/tv6Kf07VgpNDJeVlJLKpvI1+V/T+3ko4Wda717nKFf+ISkllUZJifRIWMLHt0Wwt2QFR7lSeZkmhsuwD1Gc3OJhdoRWJuqnKYSkntKhi34iJt5CfpPF6CUf8GeREkxp7tg/Sv+OVZ6nieEyIjvUJDgwgPP5A3mp82CKp55izC/TdOiiH7D3Ldy/eTEN/tzNm3c/wZkChQAdnqoUaGK4rIjGYYzrXp+wkGB2lLmFz1o/wr3blhOxZ42nQ1Muio5NJPCfkwz57XN+v7keC2q3AiBARIenKoUOV72iiMZhF74k0ttD8wR49llo2dK60Y/ySUkpqby+chY3nTvNqLb9rDv6AVnGaFJQCq0xXLvAQJgxg8xT//Br6/upMvQH3QbUR91x7giPbFzIrEad2FH6Fke59i0oZaWJ4TrEnA/hnZZ9uHP7KrptX65zG3yRMbyz6lNOFyzMOy0vbNepfQtKXaCJ4TpExybywa1diQurzegl/6PMP3/puHcfEhNv4dU+owiNW83U1n2RkiV16QulLsGlxCAiJURkiYjssv1Z/DLXZTpt0rPAqbyKiPwuIrtFZK5ttzevlZSSSla+AF6+ZxCBmRmM/3EKGKPj3n3AiJgEhs9cy7M/fMD20lX4uE47zqVn6SY8Sl2CqzWGYcAvxpjqwC+240tJNcY0sj26OpWPByYaY6oBJ4AnXIwnR9nboPeXCGPcXY9x174N9NwcSz4RbU7yYjHxFmatPcjTv39DhVPJjGr7NFn5ArS2p9RluJoYugEzbM9nYN23+ZrY9nluDdj3gb6u+z3BPrcB4ItbO7OqUgNGLPuYcil/al+DF4uOTSTs5FGe+f0bvq/VknW2fTZAZzkrdSmuJoYyxpgjtud/Apcbw1lQROJEZK2I2L/8SwIpxpgM2/FhwKvr8/a5DQEiGMnHkE6DMED0okmcO5+uO715qaSUVEYsnU6WCGPvzl4p1ZFISv3bVRODiPwsIlsv8ejmfJ2xbh59uQ2kK9n2GX0YmCQiVa83UBHpZ0succnJydd7u9tENA4jy7ZPtuWm0vy39VM0P5hA3w0/kJKarrUGLxRxbBsdd67hveY9+LNYKUe5gI5EUuoSrpoYjDFtjTH1LvH4DjgqIuUAbH8eu8xrWGx/7gWWA42Bv4EQEbFPsqsAXPZb1RgzzRgTbowJDw0NvY6P6H7OvzLnNWjH0lvCGfrrDKoct2ibtbc5f57Ry6ZxoHh5pt92n6NYgN7NKmqns1KX4GpT0gLAvjluX+C7iy8QkeIiUsD2vBTQAthuq2EsAx640v3eKNuvTBGGdXyetPyBTFg4kT+Pn/ZcYMohJt5Ci6ilvNnhWYru38Ov/V8htFQxx/DUiT0aMSaivqfDVMoruZoYooB2IrILaGs7RkTCRWS67ZraQJyIbMaaCKKMMdtt54YCg0VkN9Y+h49djCdXRDQOy7bd57GiJRnZ9mn+k/QHT67/VmdEe5h9kbzzhy0MXD2bX6rexjipSmSHmuyL6qzDU5W6CpfWSjLG/A20uUR5HPCk7flq4JI/zWxNS01cicFTRnapy/D5CaSmZwLwXZ276LhzDYNXzGTpLbcx/Mx5AP0C8oDo2ETOnU/nf4smEZiZweg2Fzbg0b8Ppa5OZz7fIOfVVwEQYUT7/pwOKsQ7C98h/Vya9jd4SFJKKv3XfMWd+zbyRtt+HChe3lGulLo6TQwusG8DKrbjvwuH8GqH56h/dA/9136lX0Qecu/ffzB45Sxi6tzJlw07Osp1aKpS10YTgxs4f+H8VLMFMXXu5PnVc7jrzCEPRpVH/fkn0TFvcaBEeV7pMMCxpLYukqfUtdPE4AbOM6IBRrZ9hhOFbmLST5MgLc2DkeUxmZnw8MMUPPMP+9//lOKlS+gieUrdAN2oxw3sXzjRsYkkpaRSpFxp9oydSPMX+sIbb8Cbb3o4wjzijTdg2TL49FNaP9SW1p6ORykfpYnBTbLt9mazf9mP3Bw1ngcOleRY3cZEdqipv1pzyuLFMGYMPPaY9aGUumHalJRDYuItPFD9AY4ULcmERRP5O/mELrSXUywW6N0b6taFqVM9HY1SPk8TQw6Jjk3kr3wFGdLpBaoetxD52xekpmfqQnvulpEBPXtCaip89RUUKuTpiJTyeZoYcoh9qOrqyo347NZ7eTxuAU0PJpCSmk7vj9Z4ODo/MmIErFwJ06ZBrVqejkYpv6CJIYc4D2Edf+djHCheluhFkyicdpZVe44zIibBg9H5iR9+gPHj4emn4eGHPR2NUn5DE0MOcR4znxpUkJfveZEKJ4/xyvJPAJj9u85xcMmBA/Doo9CoEUya5OlolPIrmhhyyMUL7W2oUIdpTe6j96afaLV3A5nmcltXqKs6fx4eesjav/DVV1CwoKcjUsqvaGLIQSO71M12PLFlH3aWrMj4H9+l2LnTNB69WEcp3YihQ2HdOvjkE6hWzdPRKOV3NDHkoIjGYbSoWsJxnJY/iJc6v0jomROM/GUaJ86mE/n1Zk0O12P+fGvT0cCB8MADV79eKXXdNDHksFlPNadPs4qO44Ry1Xnv9h7cv3UpHXauJj3T6BDWa7VnDzz+ONx2G0RHezoapfyWS4lBREqIyBIR2WX7s/glrrlbRDY5Pc6JSITt3Gciss/pXCNX4vFWYyLqO1ZgBXiveQ+2lK3GuJ/eo/Q/f+te0dfi3Dlrv0K+fDBvHgQFeToipfyWqzWGYcAvxpjqwC+242yMMcuMMY2MMY2A1sBZYLHTJZH288aYTS7G47Wch69mBORn0L0vE5yexoRFkxCTpXs3XM3gwbBxI8yYAZUrezoapfyaq4mhGzDD9nwGEHGV6x8AfjTGnHXxfX1OZIeaBOa7UG/YW7ICo9s8Rav98Twe973u3XAls2fDBx9AZCR07erpaJTye64mhjLGmCO2538CZa5yfU9g9kVlY0Vki4hMFJECLsbjtSIahxH9YEP79gAAzG7YgcXVmzH0109pmZrkueC8WWIi9OsHt98OY8d6Ohql8gQxVxlPLyI/A2UvcepVYIYxJsTp2hPGmH/1M9jOlQO2AOWNMelOZX8CQcA0YI8xZvRl7u8H9AOoWLHifw4cOHCVj+ad7BvV2/eKLnH2JLGfDqBAmdIUS4iHYN1lzOHsWWjWDJKSYNMmqFDB0xEp5dNEZIMxJvxq1121xmCMaWuMqXeJx3fAUduXu/1L/tgVXuoh4Ft7UrC99hFjlQZ8CjS5QhzTjDHhxpjw0NDQq4XttZz3ihYguHxZdkVNodieROv4fHXB88/D1q0wc6YmBaVykav7MSwA+gJRtj+/u8K1vYDhzgUiUs4Yc0REBGv/xFYX4/EJl9q7gV3rYfJk6NTJ+sjrZsywTmB79VXo2PHq1yul3OaqTUlXvFmkJDAPqAgcAB4yxhwXkXDgGWPMk7brKgOrgJuNMVlO9y8FQgEBNtnuOX219w0PDzdxcXE3HLdXOnfOOj4/ORm2bIHSpT0dkeds3QpNmkDTprBkCeTX/aSUcodrbUpyKTF4il8mBoCEBGtyaNcOFiwgW091XnH6tPW/wYkTEB8P5cp5OiKl/Ma1Jgb9KeZN6te3LiM9aBBjuwzko3rWJpTihQIZ2aWu/28Lagw88wzs3GmtKWhSUMojdEkMLxNzR3d+q3Irg2M/oupf1qW5/X1NpZh4Cy2iljK800CYNYsdTw+G1q09HZZSeZYmBi8TvWQXL90ziLOBBZjy/VsEnz8H4LdrKtmH796UuI1RP3/Ib5Ubc3+Ju/02CSrlCzQxeJmklFSSi5Tgpc6DqZl8gA+/HUuB9DQAv1xTKTo2kWLHj/J+zDhOBBdlUJeXOZthdIkQpTxIE4OXsa+ptLxqOEM6vcAd+zfx0fwxjprDS/P8q0mp6K4dfPvFy5Q8m0L/bsM5XugmAF0iRCkP0sTgZZzXVPqmfhsi7xlEiwObmTt7GKGnT5BpDIPmbvKPTX6WLuXrWUMJMFn0eHg8GyvUdpxyXnRQKZW7NDF4mYvXVPqmfhue6j6Can8f4tsvBlPr2D7A2iH94txNjIhJ8GC0Lpg1Czp2JKtCBXo9PpHtZW5xnAoODMi2Z7ZSKndpYvBCEY3DmPhQI4IDAwBYWq0JDz48noCsLL6ZGUmHnasBMMCstQd9q+ZgDIwbB336QIsWFItby8DH2ziWCAkLCWZc9/r+PzRXKS+mE9y8WEy8hZfmbSbT9ndU+p+/mfbtWBod2cn7zR7g7ZaPkJkvgLCQYFYN84HhnRkZ1vWP/vc/6NULPv0UCvjtgrpKeR23LaKnPCeicRhvP9TQUXM4VrQkPR6O4suGHem/9mu+mPsapc6c8I2O2jNnoHt3a1IYOtS6MJ4mBaW8ks589nL2JpVRC7aRkppOWv4gXuk4gI1htRiz+H1++OwFRvUeSUy8hejYRJJSUikfEkxkh5oebY5xjqdO/nPMjPkvxXckwNSp0L+/x+JSSl2d1hh8QETjMDaNbE+fZhUde0d/Xb8t9z0ygbTAAkyd/hLbIkdhOXEWA1hSUon8ynPDWu2T1iwpqVQ+buH9qQMomPgHayd8pElBKR+gNQYfMiaiPuGVSjh+iZ+qUZetMT+z5+mnePWX6TQ6vIOhnV7gdIFCpGcZXpxr3UI7t2oO9lqCxda0devhHUyf/18M0KvXmySnVWZVrkSilHKFdj77gcpDf6DfuvkM+XUGB4qX55mI4ewKrZTtmrAcbl5y3pmuaNoZBq+YyaMbF3LopjL0fegNDhQvjwD7ojrnyPsrpa5OV1fNS0SY1vR+tpSrzpQFb/HdF4OZ0PJRZja+h/P5AwFr89Lw+dY5D+5KDtZksIXUdOsWG/myMum+fTnDln9GqTMpzGrciehWj3KqYBFAJ60p5Ss0MfiB4oUCOXE2nbUVG9C572Sif5zM60s/4v/iYph4Rx++q3MnGQH5SU3PJDo20aXEMCImgdm/H3IMoQXAGDrsWsNLv82kxt8H2Vy2Ok/e/xpbytVwXKKT1pTyHa7u4PYgMAqoDTQxxlyyfUdEOgKTgQBgujEmylZeBZgDlAQ2AI8YY85f7X21KSm7mHgLg2z9CXYt9m9i6K+f0eDP3fwTFMzqSg35rcqt/FblVlb874lrek3nUU531wrlmw2HHbUDgMrHLbTftZb7ti2jdvJ+9pSowNst+/BjzdsxcmFcQ043Yymlrk2u7OAmIrWBLOBD4OVLJQYRCQB2Au2Aw8B6oJcxZruIzAPmG2PmiMj/gM3GmA+u9r6aGP5tREwCM9cezF5oDHfvjaPdrt9ptW8jFU4ds5bXqAEdOlgfd90FhQsTE29xDIm9nBJnT9IoKZGmh7Zy9544avxtfb/NZavzxa2d+bbu3WTmC3BcHxwYoLOYlfIiubq1p4gs5/KJoTkwyhjTwXY83HYqCkgGyhpjMi6+7ko0MVzaFb/cjaHOqSNEFUmiwbbfYflySE2FoCCONWrCp4VrsKxyY/4IrUyAyaLSiSNU/+sgdY/uoe6xvdQ9uoeyp48DkBaQnw1hdVhcvRlLqjfDctO/96fWWoJS3sebOp/DgENOx4eBplibj1KMMRlO5fot4oKIxmHZvoizNQcVL0S/nvfQwH7+3DlYsQJiY/ln5nyGHl3JUOCfoGCCMtMpkGn9a8mUfOwuWYHVlRqyrfQtJJSrzuay1UkLvPys5T7NKjImon5OflSlVA66amIQkZ+Bspc49aox5jv3h3TZOPoB/QAqVqyYW2/r0y5OFNkULAjt2kG7drTNfzdlTv1Fy/0bqXNsH+fyF2BXqZvZVbIiO0tVvGIScJZP4OGmmhSU8nVXTQzGmLYuvocFuNnpuIKt7G8gRETy22oN9vLLxTENmAbWpiQXY1JOyocEY6EUXzVof133aSJQyj/lxpIY64HqIlJFRIKAnsACY+3cWAY8YLuuL5BrNRB1gfPmQJciQIuqJbItjT2pRyP2juusSUEpP+RSH4OI3AdMAUKBhSKyyRjTQUTKYx2Weo+tY3kAEIt1uOonxhj7rvZDgTkiMgaIBz52JR51Yy5eqA+stYEso53ISuVFuiSGUkrlEbofg1JKqRuiiUEppVQ2mhiUUkplo4lBKaVUNpoYlFJKZaOJQSmlVDY+OVxVRJKBAzd4eyngLzeG42n+9Hn0s3gnf/os4F+f53o/SyVjTOjVLvLJxOAKEYm7lnG8vsKfPo9+Fu/kT58F/Ovz5NRn0aYkpZRS2WhiUEoplU1eTAzTPB2Am/nT59HP4p386bOAf32eHPksea6PQSml1JXlxRqDUkqpK8iTiUFE/isiW0Rkk4gsti0T7pNEJFpE/rB9nm9FJMTTMblCRB4UkW0ikiUiPjlyREQ6ikiiiOwWkWGejudGicgnInJMRLZ6OhZXicjNIrJMRLbb/n294OmYXCEiBUVknYhstn2eN9z6+nmxKUlEihljTtmeDwTqGGOe8XBYN0RE2gNLbftejAcwxgz1cFg3TERqA1nAh8DLxhifWl9dRAKAnUA7rPuYrwd6GWO2ezSwGyAirYDTwOfGmHqejscVIlIOKGeM2SgiRYENQIQv/r0AiIgAhY0xp0UkEFgJvGCMWeuO18+TNQZ7UrApDPhsdjTGLLZtjQqwFusWqT7LGLPDGJPo6Thc0ATYbYzZa4w5D8wBunk4phtijPkNOO7pONzBGHPEGLPR9vwfYAfgs7tPGavTtsNA28Nt32N5MjEAiMhYETkE9AZe93Q8bvJ/wI+eDiKPCwMOOR0fxoe/gPyRiFQGGgO/ezYS14hIgIhsAo4BS4wxbvs8fpsYRORnEdl6iUc3AGPMq8aYm4FZwADPRntlV/sstmteBTKwfh6vdi2fR6mcICJFgG+AQRe1HPgcY0ymMaYR1laCJiLituY+l/Z89mbGmLbXeOksYBEwMgfDccnVPouIPAbcC7QxPtBpdB1/N77IAtzsdFzBVqY8zNYW/w0wyxgz39PxuIsxJkVElgEdAbcMFPDbGsOViEh1p8NuwB+eisVVItIRGAJ0Ncac9XQ8ivVAdRGpIiJBQE9ggYdjyvNsnbUfAzuMMe94Oh5XiUiofQSiiARjHezgtu+xvDoq6RugJtbRLweAZ4wxPvmrTkR2AwWAv21Fa311hBWAiNwHTAFCgRRgkzGmg2ejuj4icg8wCQgAPjHGjPVwSDdERGYDd2FdwfMoMNIY87FHg7pBInIHsAJIwPr/PcArxphFnovqxolIA2AG1n9j+YB5xpjRbnv9vJgYlFJKXV6ebEpSSil1eZoYlFJKZaOJQSmlVDaaGJRSSmWjiUEppVQ2mhiUUkplo4lBKaVUNpoYlFJKZfP/XhRe+ScHbB4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f2f6c6bc0d0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# \"hidden_layers\" specifies the number of neurons per layer.\n",
    "# The network will be trained with batch_size x \"steps\" random datapoints.\n",
    "\n",
    "### YOUR ACTION REQUIRED:\n",
    "# Try to achieve a good function approximation by finding good parameters...\n",
    "hidden_layers, steps = [100, 50], 10000 #hidden_layers, steps = [5, 5, 5, 5, 5], 1000\n",
    "\n",
    "tf_sin = FunctionApproximator(f=np.sin, hidden_layers=hidden_layers)\n",
    "\n",
    "# Train the network. Note that we do not directly provide x and f(x) samples,\n",
    "# but rather the function itself and .fit() will then generate samples from\n",
    "# the function (unlike scikit learn's .fit() method...)\n",
    "losses, dt = tf_sin.fit(x_min=-3., x_max=3., steps=steps, batch_size=1000,\n",
    "                        show_progress=True)\n",
    "print '%.2f seconds' % dt\n",
    "\n",
    "# Visualize some predictions\n",
    "x_data = np.random.uniform(low=-3., high=3., size=100)\n",
    "x_data.sort()\n",
    "y_data = np.sin(x_data)\n",
    "y_predictions = tf_sin.predict(x_data)\n",
    "\n",
    "pyplot.scatter(x_data, y_data)\n",
    "pyplot.plot(x_data, y_predictions, 'r-')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A References\n",
    "\n",
    "Unfortunately, we don't have time to talk about more ML in this\n",
    "workshop. Two good online references:\n",
    "\n",
    "- http://deeplearningbook.org (by Ian Goodfellow et al) – more theoretical\n",
    "- http://neuralnetworksanddeeplearning.com/ (by Michael Nielson) – more hands-on"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
